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PREFACE 


Part  of  the  Rome  Air  Development  Center's  (RADC)  speech 
research  is  an  in-house  effort  to  develop  vector  classification 
concepts  for  recognition  problems  in  general  and  for  speech 
recognition  in  particular.  These  concepts,  referred  to  as 
Mathematical  Intelligence  (MI),  were  developed  by  Capt  William  I. 
Lundgren  prior  to  his  departure  from  RADC  in  October  1985.  This 
report  is  designed  to  familiarize  the  reader  with  MI,  show  how  it 
applies  to  speech  recognition,  and  discuss  the  preliminary  test 
results. 

The  MI  system  developed  at  RADC  consists  of  numerous  programs 
that  were  written  to  run  on  a  DEC  PDP  11/70  minicomputer  and  an  FPS 
5210  floating  point  array  processor.  This  configuration  is 
discussed  in  more  detail  in  Chapter  3.  Many  of  the  programs 
implement  a  rudimentary  digital  speech  processing  tool  (DSPT)  that 
performs  such  functions  as  digital  record  and  playback  of  speech, 
speech  spectrogram  display,  and  symbolic  labeling  of  spectrogram 
segments.  This  tool  is  useful  for  observing  speech  data  graphically 
to  build  the  MI  structure  manually  as  discussed  in  Chapter  2.  The 
remainder  of  the  programs  implement  MI  speech  recognition.  At  this 
time,  the  DSPT  is  functional  and  available  for  use.  The  MI  speech 
recognizer  is  not. 
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The  numerous  MI  programs  developed  at  RADC  are,  for  the  most 
part,  undocumented.  In  addition,  the  algorithms  used  to  write  the 
programs  are  also  undocumented.  For  example,  there  is  no 
documentation  on  the  windowing  algorithm  used  in  the  DSPT.  The 
author  believes,  however,  that  the  MI  concepts  are  sufficiently 
documented  in  this  report  to  allow  any  speech  researcher  to 
implement  an  MI  development  system  (Chapter  2). 
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1.  Introduction 


The  goal  of  speech  recognition  is  to  provide  a  more  natural  way 


for  humans  to  enter  commands  and  data  to  a  computer.  In  some  cases. 


this  is  a  convenience.  In  other  cases,  where  a  human's  hands  and/or 


eyes  are  already  busy,  speech  input  is  a  necessity.  The  success  of 


applying  current  speech  recognition  technology  is  largely  dependent 


upon  the  particular  application  because  human-like  performance  has 


not  yet  been  achieved.  Some  of  the  key  issues  in  speech  recognition 


performance  are  listed  in  Table  1-1.  Also  shown  is  how  these  issues 


compare  between  a  fragile  system  and  a  robust  system.  A  fragile 


system's  performance  deteriorates  quickly  when  strict  constraints  are 


not  maintained.  A  robust  system  requires  fewer  constraints.  Humans 


are  the  most  robust  speech  recognition  system  available  today. 


Two  of  the  most  popular  techniques  currently  used  for  speech 


recognition  are  Dynamic  Time  Warping  (DTW)  and  Hidden  Markov 


Modeling  (HMM).  A  detailed  discussion  of  these  techniques  is  beyond 


the  scope  of  this  report.  However,  the  reader  who  is  not  familiar 


with  these  techniques  can  consult  the  references.  Since  these 


techniques  have  not  produced  human-like  performance,  further 


research  and  development  are  required  to  both  modify  these 


techniques  and  to  investigate  unique  alternatives.  MI  falls  into 


the  later  category. 
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Table  1-1  Speech  Recognition  Issues 


Issue 

Fragile  System 

Robust  System 

Continuous/ 

Isolated 

Speech 

Isolated  and 

Limited  Connected 

Continuous 

Intraspeaker 

Variability 

-  Stress 

-  Inflections 

-  Enunciation 

-  Vocal  Tract 
Changes  (e.g., 
Illness) 

Increase  in 

Recognition  Errors 

Accurate 

Recognition 

Interspeaker 

Variability 

-  Male/Female 

-  Dialect 

Increase  in 

Recognition  Errors 

Accurate 

Recognition 

Environmental 

Variabi 1 ity 

-  Transducer 
Response 

-  Noise 

-  Acoustic 

Chamber 

Increase  in 

Recognition  Errors 

Accurate 

Recognition 

Vocabulary  Size 

Smal  1 

Large 

Speech 

Understanding 

None  or  Syntactic 

Only 

Syntactic, Semantii 
Pragmatic,  &  Goal 
Understanding 

Computational 

Power  Required 

Smal  1 

Large 

2.  Mathematical  Intelligence 


Both  methods  of  recognition  mentioned  in  the  previous  chapter 
attempt  to  map  similar  inputs  into  a  single  class  so  a  particular 
class  can  be  recognized.  Similarity  is  measured  between  expected 
(training)  data  and  actual  (recognition)  data.  Various  methods  are 
used  to  modify  the  input  so  that  it  will  be  similar  enough  to  match 
at  least  one  class.  If  the  inputs  are  not  similar  enough, 
recognition  will  not  occur.  With  this  approach,  all  of  the  input 
information  is  used  to  determine  similarity  of  the  inputs  and 
subsequent  vector  classification.  The  key  distinguishing  feature  of 
Mathematical  Intelligence  (MI)  is  that  information  in  the  input  not 
relevant  to  the  recognition  is  filtered  out  by  means  of  a 
mathematical  structure.  In  other  words,  not  all  of  the  input 
information  is  used  to  classify  a  particular  vector. 

2.1.  Vector  Classification  Concepts 

This  section  describes  how  MI  maps  an  input  feature  vector  into 
its  appropriate  class  (Figure  2.1).  The  input  feature  vector  is 
derived  from  pre-determined  mathematical  operations  on  an  input 
analog  signal.  The  mathematical  operations  result  in  parameters 
that  describe  various  features  of  the  analog  signal.  These  feature 
vectors  are  sequenced  in  time  so  that  each  feature  vector  describes 
the  analog  signal  for  some  specific  period  of  time.  That  period  of 
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time  is  called  a  frame  or  window.  A  feature  vector  can  contain 


features  that  span  more  than  one  window. 

Once  a  feature  vector  is  determined,  the  magnitudes  of  the 
features  within  that  vector  are  compared  to  each  other.  This  is 
accomplished  using  pre-defined  pseudo-logical  functions.  For 
example,  let  the  values  of  features  A,  B,  C,  and  D  be  0.8,  0.95, 
0.2,  and  0.54,  respectively.  Then  a  function,  FI,  can  be  described 
as  "A  and  B  are  large  compared  to  C."  MI  obtains  this  example 
relationship  mathematically  by 

A  x  B 
Fl  = - 

1  +  C 

The  number  and  type  of  functions  used  is  determined  by  what  is 
required  to  separate  an  input  feature  vector  into  its  class  and  by 
the  limit  of  information  contained  in  that  vector.  The  combined  set 
of  pre-defined  functions  forms  the  function  vector  which  is  applied 
to  each  class. 

For  each  class,  there  is  a  Threshold  Vector  where  each 
component  corresponds  to  a  component  in  the  Function  Vector.  If  the 
function  component  is  greater  than  the  corresponding  threshold 
component,  then  a  value  of  one  is  assigned;  otherwise  a  value  of 
zero  is  assigned.  The  result  of  this  sequence  of  operations  is  a 
pre-output  vector  for  each  class  composed  of  a  set  of  ones  and 
zeros. 
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Next,  the  ones  and  zeros  in  each  Pre-output  Vector  are  summed 
to  obtain  an  integer  value  corresponding  to  each  class.  The  class 
with  the  largest  integer  value  is  the  selected  class  for  the  input 
feature  vector. 

The  MI  vector  classification  procedure  can  be  described 
mathematically  as  follows: 

Pm  =  TH(F1, Tim) ,  TH(F2,T2m),  .....  TH(Fn,Tnm) 

Class  =  NUM  (MAX  (11,12,..., Im) ) 
where 

P  =  pre-output  vector  (one  for  each  class). 

in  =  class  number. 

TH  =  threshold  function;  results  in  a  one  if  the  first 
argument  is  greater  than  the  second, 
otherwise  results  in  a  zero. 

F  =  function. 

n  =  function  number. 

T  -  threshold. 


ANALOG  SIGNAL 


Figure  2.1  Mapping  of  an  Input  Feature  Vector  into  its  Class 
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2.2.  Vector  Classification  Example 


Suppose  there  is  an  Input  Feature  Vector  with  four  features,  and 
the  objective  is  to  map  it  into  one  of  six  classes.  The  following 
are  given: 

Input  Feature  Vector  =  (A,  B,  C,  D) 

Feature  Values:  A  =  0.25,  B  =  0.87,  C  =  0.79,  D  =  0.62 
Function  Vector  =  (FI,  F2,  F3,  F4,  F5) 

Functions : 


A  x  B  x  C 

Fi  =  - 

1  +  D 


A  x  D 

F2  -  . . 

(1  +  B)  (1  +  C) 

B  x  C 

F3  =  . . 

(1  +  A)  (1  +  D) 

C 

F4  =  . . . . - 

(1  +  A)  (1  +  B)  (1  +  D) 

B 

F5  =  - - - 

(1  +  A)  (1  +  C)  (1  +  D) 

Threshold  Vectors: 


Class 

1: 

(.33, 

1.0, 

•  41, 

1.0, 

1.0) 

Class 

2: 

(i.o. 

.25, 

1.0, 

.39, 

1.0) 

Class 

3: 

(1.0, 

1 . 0 , 

.27, 

1.0, 

.43) 

Class 

4: 

(.44, 

1.0, 

1.0, 

.36, 

1.0) 

Class 

5: 

(i.o. 

.35, 

i.o. 

i.o, 

.26) 
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Class  6:  (1.0,  1.0,  .35,  .42,  1.0) 


By  applying  the  functions  of  the  Function  Vector  to  the  Input 
Feature  Vector,  the  Evaluated  Function  Vector  (EFV)  obtained  is 
EFV  =  (.11,  .05,  .34,  .21,  .24). 

Next,  a  Pre-Output  Vector  is  calculated  for  each  class.  For  Class 

1, 

PI  =  (TH(F1,T11),  TH(F2,T21),  TH(F3,T31),  TH(F4,T41), 

TH(F5,T51 ) ) 

=  (TH( .11, .33) ,  TH( .05, 1 .0) ,  TH( .34, .41) ,  TH(.21,1.0), 

TH( . 24,1 .0) ) 

=  (0,  0,  0,  0,  0) 

Thus, 

II  =  0. 

Similarly,  for  Class  2-6, 


P2  = 

(0, 

0, 

0, 

0, 

0) 

12 

=  0 

P3  = 

(0, 

0, 

1, 

0, 

0) 

13 

=  1 

P4  = 

(0, 

0, 

0, 

0, 

0) 

14 

=  0 

P5  = 

(0, 

0, 

0, 

0, 

0) 

15 

=  0 

P6  = 

(0, 

0, 

0, 

0, 

0) 

16 

=  0 

Then, 
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Class  =  NUM  (MAX  (II,  12,  13,  14,  15,  16)) 
=  NUM  (MAX  (0,  0,  1,  0,  0,  0)) 


Thus,  the  Input  Feature  Vector,  (.25,  .87,  .79,  .62)  belongs  to 
Class  3.  This  procedure  is  repeated  for  every  Input  Feature  Vector. 


2.3.  Speech  Recognition 

The  MI  vector  classification  procedure  is  directly  applicable 
to  speech  recognition.  In  this  case,  the  analog  signal  (Figure  2.1) 
is  transduced  speech.  The  speech  is  digitized  and  various 
parameters  are  extracted  to  construct  the  Input  Feature  Vector 
(IFV).  Features  which  might  be  considered  include  the  Power 
Spectral  Density  for  a  given  set  of  frequency  bands,  pitch  period, 
formant  frequencies,  etc.  The  reader  can  consult  the  references  for 
a  more  detailed  discussion  of  speech  features. 

Recognition  occurs  when  the  IFV(s)  is  mapped  into  its 
appropriate  class.  Thus,  there  is  one  class  or  sequence  of  classes 
that  corresponds  to  each  speech  segment  that  the  system  has  been 
trained  to  recognize. 

2.4  Training 

Training  an  MI  speech  recognizer  to  recognize  speech  segments 
involves  constructing  the  Function  Vector  for  all  classes  and  a 
Threshold  Vector  for  each  class.  In  the  example  of  MI  vector 


classification  presented  in  an  earlier  section,  the  Function  Vector 
and  the  Threshold  Vectors  were  simply  given.  However,  these  vectors 
must  somehow  be  derived  by  training  the  system.  This  can  be  done 
manually  by  the  user  or  automatically  by  the  system. 

It  is  crucial  in  the  training  process  that  valid  selection 
criteria  are  used  to  select  the  Function  Vector  and  Threshold 
Vectors  that  will  be  used  for  recognition.  Unfortunately,  the 
author  is  not  familiar  with  the  criteria  that  were  used  in 
implementing  the  MI  system  at  RADC,  nor  the  results  of  testing  those 
criteria. 

2.4.1.  Manual  Structure  Building 

In  order  for  the  user  to  manually  build  an  MI  structure,  he 
must  be  able  to  isolate  the  speech  segment  to  be  recognized  and 
observe  the  sequence  of  Feature  Vectors  that  represent  that  segment. 
The  user  can  then  determine  the  relationships  between  the  normalized 
values  (from  0  to  1)  of  the  features,  based  on  some  selection 
criteria. 

For  example,  suppose  the  problem  is  to  recognize  several  spoken 
words,  and  the  training  vocabulary  consists  of  the  same  words.  An 


algorithm  could  be  employed  to  calculate  the  power  spectral  density 
(PSD)  values  for  a  spectrum  divided  logarithmically  into  32 
frequencv  bands.  Each  Feature  Vector  then  would  have  32  features. 


I; 

I; 


These  features  can  be  displayed  as  a  spectrogram  on  a  high- 


resolution  color  graphics  device.  The  color  represents  the  PSD  of 


each  feature  over  a  single  frame.  The  user  observes  the  colors  to 


compare  relative  PSD  between  features  and  derives  functions  and 


Threshold  Vectors  that  describe  the  relationships.  The  choice  of 


functions  and  Threshold  Vectors  must  be  such  that  each  word  in  the 


training  vocabulary  is  unique. 


2.4.2.  Automatic  Generation 


In  the  example  presented  in  the  previous  section,  the  user  was 


required  to  select  functions  and  Threshold  Vectors.  For  a 


vocabulary  of  just  a  few  words  (or  other  speech  segments)  the 


selection  criteria  need  not  be  well-defined  as  long  as  acceptable 


recognition  performance  is  achieved.  However,  for  a  vocabulary  of 


practical  size  (100+  words),  the  selection  criteria  must  be  highly 


definitive,  or  the  user  will  spend  endless  hours  "tweaking"  the 


system  to  achieve  acceptable  recognition  performance.  If  the 


selection  criteria  can  be  well-defined,  then  it's  also  possible  to 


automate  the  selection  of  functions  and  Threshold  Vectors.  This  is 


the  essence  of  the  automatic  generation  process. 


2.5.  MI  Development  System 


In  order  to  develop  an  MI  speech  recognizer  within  reasonable 
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time,  cost  and  manpower  constraints,  a  complete  development  system 
is  required.  The  essential  components  and  their  interrelationships 
are  illustrated  in  Figure  2.2.  The  central  component  is  the  Digital 
Speech  Processing  Tool  (DSPT). 

The  DSPT  interacts  directly  with  the  user  interface  and  with 
all  other  components  of  the  MI  development  system.  It  has 
facilities  for  evaluating  recognition  results,  developing  DSP 
algorithms,  training  the  system  either  manually  or  automatically, 
and  supplying  both  raw  and  derived  data  to  the  user  interface.  DSP 
algorithms,  the  Function  Vector,  and  the  Threshold  Vectors  originate 
in  the  DSPT  and  are  then  down-loaded  into  the  MI  recognizer.  DSPT 
capabilities  are  summarized  in  Table  2-1. 


TABLE  2-1  DSPT  Capabilities 


Digital  Record/Playback 

Provide  Raw  Data  to  User 
Interface 

Window  Type 

Window  Overlap 

High  Pass  F i  Iter 

Band  Pass  Fi  Iter 

Sample  Quantization 

Various  Display  Options 


DSP  Feature  Extraction 

Provide  Feature  Data  to 
User  Interface 

Window  Length 

Spectral  Shaping 

Low  Pass  Fi Iter 

Sampling  Rate 

Length  of  Speech  Segment 

DSP  Algorithm 
Development 
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The  Speech  Data  Base  consists  of  digitally  recorded  audio 
correlated  with  the  symbolic  representation  (e.g.,  phonemes,  words) 
of  the  audio.  This  is  referred  to  as  a  "labeled"  speech  data  base. 
Having  correlated  audio  data  and  symbolic  data  allows  for  a  direct 
automated  evaluation  of  the  recognition  results.  It  also  allows 
the  automatic  generation  algorithm  to  train  the  system  without  user 
intervention. 

The  User  Interface  consists  of  various  Input/Output  devices 
that  link  the  user  to  the  DSPT  and  the  MI  Speech  Recognizer.  These 
include  microphones,  speakers,  amplifiers,  textual  display,  graphics 
display,  keyboard,  etc. 

2.6.  Unresolved  Difficulties 

The  theoretical  basis  for  MI  speech  recognition  can  be  found  in 
two  of  the  references  (Lundgren  84,  Lundgren  85).  Although  it  is 
not  based  on  any  well-recognized  model  of  human  aural  cognition,  it 
can  still  be  useful.  The  author  has  identified  several  possible 
problems  that  should  receive  additional  analysis  and  that  may  need 
to  be  resolved  to  make  the  MI  method  more  useful. 

2.6.1.  Selecting  Largest  Integer 

Part  of  the  example  of  MI  vector  classification  presented  in  an 
earlier  section  was  to  determine  the  integer  sums  (one  for  each  class), 
determine  the  maximum  of  the  integer  sums,  and  select  the  class  cor¬ 
responding  to  the  maximum  integer  sum.  A  conflict  can  arise  if  the 
maximum  integer  sum  value  belongs  to  more  than  one  class.  Of  course, 
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Figure  2.2  MI  Speech  Recognition  Development  System 
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the  Function  Vector  and  Threshold  Vectors  constructed  during  training 
should  minimize  the  possibility  of  such  a  conflict.  Nevertheless,  it 
is  possible  and  should  be  dealt  with.  The  author  is  not  a  ^re  of  whethe 
or  not  this  issue  was  addressed  in  the  RADC  MI  system. 

2.6.2.  Non-Linear  Functions 

When  a  particular  feature  of  an  Input  Feature  Vector  is  used  in 
the  numerator  of  a  function,  the  calculated  function  value  will 
change  linearly  with  respect  to  changes  in  the  feature  value. 
However,  if  the  feature  is  used  in  the  denominator  of  a  function, 
the  calculated  function  will  change  non-linearly  with  respect  to 
changes  in  the  feature  value.  The  author  does  not  know  why  this 
dichotomy  of  functional  behavior  was  implemented  or  whether  it  poses 
a  problem  in  the  recognition  process. 

Consider  as  an  example  a  function,  Fl=(AxB)/C  and  a  sequence  of 
three  Input  Feature  Vectors, 

(A  =  0.85,  B  =  0.75,  C  =  0.20) 

(A  =  0.85,  B  =  0.75,  C  =  0.40) 

(A  -  0.85,  B  =  0.75,  C  =  0.80). 

The  calculated  values  for  FI  are  0.53,  0.46  and  0.35,  respectively. 

While  the  value  of  C  increased  by  a  factor  of  two  in  each  case 
(vector  1  to  2,  vector  2  to  3),  the  value  of  FI  did  not  decrease  by 
a  factor  of  two.  In  fact,  FI  first  decreased  by  a  factor  of  0.87 
and  next  by  a  factor  of  0.76,  indicating  a  non-linear  relationship 


between  the  value  of  C  and  FI. 
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Suppose  the  values  of  an  Input  Feature  Vector  are  (A  =  0.75, 

B  =  0.87,  C  =  0.98,  D  =  0.77).  Two  valid  functions  that  could  be 
constructed  and  correspond  to  the  feature  values  are  Fl=AxBxCxD  and 
F2=AxB.  Intuitively,  it  seems  because  both  functions  apply,  and 
because  FI  uses  more  information  from  the  Input  Feature  Vector  than 
F2,  that  the  calculated  value  of  FI  would  exceed  F2.  In  other 
words,  one  should  have  more  confidence  in  FI  and  F2.  Yet  just  the 
opposite  is  true.  The  confidence  that  A,  B,  C,  and  D  are  large  is 
0.49  for  FI  and  0.65  for  F2,  even  though  F2  considered  only  two 
features!  This  reverse  logic  says,  in  effect,  "to  gain  the  highest 
confidence  that  a  particular  condition  exists,  use  the  least  amount 
of  information." 


JJjl, 


3.  Initial  Testing  of  MI 


Although  speech  recognition  using  MI  has  not  been  formally 


tested,  some  informal  testing  has  been  done.  More  specifically, 


pitch  detection  was  tested  using  MI.  The  problem  is  to  correctly 


classify  the  pitch  frequency  of  a  speaker's  voice  within  each  10  Hz 


class  for  the  duration  of  the  speech  utterance  examined.  The 


automatic  generation  algorithm  was  not  used  so  that  the  MI  structure 


itself  and  not  the  automatic  generation  could  be  evaluated. 


3.1  System  Configuration 


Figure  3.1  is  a  block  diagram  of  the  basic  system  used  to 


develop  the  MI  software  and  perform  the  initial  testing. 


operating  system  was  RSX-llM  by  Digital  Equipment  Corporation  (DEC). 


The  software  run  on  ^.he  PDP-11  was  written  in  Fortran  77.  Software 


was  also  written  to  run  on  the  Floating  Point  Systems  (EPS)  5210 


array  processor. 


3.2  Pitch  Detection 


The  input  data  to  the  pitch  recognition  system  was  generated  by 


sampling  speech  at  a  rate  of  10  kHz  and  using  a  Fast  Fourier 


Transform  (FFT)  on  1024  samples  to  generate  a  512-point  Power 


Spectral  Density  (PSD).  This  corresponds  to  examining  sneech 


segments  102.4  msec  in  length.  A  Hanning  window  was  used  wild  a 


weighting  of  6  dB/octave  above  500  Hz.  The  pitch  information  is 


A'WiCi'W 


contained  in  pitch  peaks  that  are  located  in  the  fundamental  (the 
pitch  itself)  and  the  harmonics  (integral  multiples  of  the  pitch). 
Informal  test  results  indicated  that  the  MI  system  identified  the 
correct  pitch  about  90%  of  the  time. 
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Figure  3.1  Block  diagram  of  the  MI  development  system  hardware 
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4.  Summary  and  Conclusions 


This  report  has  discussed  the  problem  of  speech  recognition  and 
how  MI  can  be  employed  to  solve  the  problem.  The  basic  MI  concepts 
were  described  in  enough  detail  to  allow  a  speech  researcher  to 
implement  one  form  of  a  MI  system.  Much  research  is  necessary  to 
permit  a  meaningful  comparison  of  MI  to  other  speech  recognition 
techniques.  This  will  require  a  wel 1-integrated  MI  development 
system  as  described  earlier  and  a  library  of  popular  speech  data 
bases  used  to  evaluate  speech  recognition.  Currently,  only  a 
partial  development  system  exists  at  RADC  and  there  are  no  plans  to 
extend  it  beyond  its  present  form. 
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